calculate the sleep interval differently instead

Right now, the base of the exponent is 2. This means that for any downtime of X hours, the maximum time until the federation resumes is another X hours - basically at maximum doubling the downtime. The expected / average value should be half the downtime I think (due to discretization).

Maybe just changing the constant to e.g. 1.5 or 1.25 would be good? That would mean at maximum the time to resume would be 1/2 or 1/4 of the downtime.

Here's a simulation of downtimes with different exponential delays:

2**retries (current in lemmy)

1.5**retries

1.25**retries

code to generate these tables

Also I was actually thinking that it might be smart to go the opposite direction than this PR and remove the "dead" flag altogether and instead just keep exponentially increasing the retry delays for all instances - maybe with a limit of a week or so.

Capping at an hour I guess kinda works, then we bascially have an exponential delay between 0s and 1h, then fixed at every 1h for a day, and after that fixed at 24h (the dead check). But reducing the base of the exponentiation seems cleaner.

I noticed another problem with the outgoing federation workers. It looks like the list of dead and alive instances is never reloaded after starting.

I don't think that's true, the list of instances is fully refreshed in the loop in start_stop_federation_workers, which happens every INSTANCES_RECHECK_DELAY (once per minute)

Also I was actually thinking that it might be smart to go the opposite direction than this PR and remove the "dead" flag altogether and instead just keep exponentially increasing the retry delays for all instances - maybe with a limit of a week or so.

That seems like a good idea, it means there is one less value to check and less complexity. However I dont like the values you are showing. If an instance is down for 3 hours it will be defederated for 6.5 hours before federation resumes. That doesnt make sense because its extremely cheap to send an inbox request, so we can easily do it once per hour during the entire first day. For the limit one day should be fine, its what we are using now as well.

I don't think that's true, the list of instances is fully refreshed in the loop in start_stop_federation_workers, which happens every INSTANCES_RECHECK_DELAY (once per minute)

True, I missed the loop there.

If an instance is down for 3 hours it will be defederated for 6.5 hours

I think you might be misreading my tables (they are a bit confusing), the second column is the total duration including the downtime, so for the 3 h downtime the additional delay until federation resumes would be:

With 1.25^N the delay until refederation is around or less than your fixed 1h limit all the way up to an instance downtime of > 24h

Okay in that case it sounds good, Ive changed it to 1.25^n. And I also changed it to ignore the first error, so it doesnt sleep over a second with only a single failure. Instead there need to be at least two failures before it starts sleeping.

I also thought about getting not marking instances as dead anymore. But then we would have to keep thousands of federation workers around for dead instances which do nothing but sleep for a very long time. We could instead mark instances as dead based on last_successful_published_time, but that isnt available in start_stop_federation_workers() so it would be unnecessarily complicated.